Image processing apparatus, image pickup apparatus, and image processing method

ABSTRACT

An apparatus includes an acquisition unit configured to acquire first polarization information that includes a polarized light component and an angle component using luminance values of input images obtained by imaging light of a plurality of colors in a plurality of different polarization states, a correction unit configured to acquire corrected polarization information obtained by replacing at least part of the first polarization information in a specific area with second polarization information obtained using a luminance value of a specific color among the plurality of colors in the specific area in the input images, and a generation unit configured to generate an output image using the corrected polarization information. The polarized light component is a luminance component that changes according to polarization angles. The angle component is a polarization angle that maximizes the luminance value.

BACKGROUND OF THE DISCLOSURE Field of the Disclosure

The aspect of the embodiments relates to image processing technologiesfor a plurality of colored images including polarization information.

Description of the Related Art

It is known that a predetermined feature of an object can be emphasizedand detected by observing a polarization state of light from the object.For example, changing a polarization direction of light that passesthrough a polarizing filter mounted in front of the lens in asingle-lens reflex camera and capturing an image can provide an effectof highlighting a texture such as a color and contrast of an object andan effect of emphasizing or reducing the reflection of reflected lighton a water surface or the like.

Japanese Patent Laid-Open No. (“JP”) 2017-191986 discloses a method forreducing coloring in a generated output image in calculatingpolarization information from a plurality of input images generated byimaging with different polarization states and by generating an outputimage from the polarization information. More specifically, this methodcorrects coloring of the combined image based on the color informationof the reference image selected from the plurality of input images.

The method disclosed in JP 2017-191986 has an issue in that the coloringof the output image is not corrected when the color information of thereference image is not correctly acquired.

SUMMARY OF THE DISCLOSURE

An apparatus according to one aspect of the embodiments includes atleast one processor and a memory coupled to the at least one processor,the memory having instructions that, when executed by the processor,performs operations as an acquisition unit configured to acquire firstpolarization information that includes a polarized light component andan angle component using luminance values of input images obtained byimaging light of a plurality of colors in a plurality of differentpolarization states, a correction unit configured to acquire correctedpolarization information obtained by replacing at least part of thefirst polarization information in a specific area with secondpolarization information obtained using a luminance value of a specificcolor among the plurality of colors in the specific area in the inputimages, and a generation unit configured to generate an output imageusing the corrected polarization information. The polarized lightcomponent is a luminance component that changes according topolarization angles. The angle component is a polarization angle thatmaximizes the luminance value.

An apparatus according to another aspect of the embodiments includes aprocessor and a memory coupled to the at least one processor, the memoryhaving instructions that, when executed by the processor, performsoperations as an acquisition unit configured to acquire a plurality ofinput images generated with a plurality of polarized light beams havingdifferent polarization states from each other via a variable retardationplate, and to acquire polarization information at a representativewavelength of each of the plurality of input images, a setting unitconfigured to set the representative wavelength for each of a pluralityof color channels in the plurality of input images, and a generationunit configured to generate an output image from the polarizationinformation. The setting unit acquires information on a polarizationangle for each color channel in the plurality of input images from therepresentative wavelength, acquires information on a luminance and phasethat changes according to the polarization angle for each color channel,from the information on the polarization angle and the plurality ofinput images, and sets the representative wavelength for each colorchannel according to a phase difference between the color channels.

An image pickup apparatus including the above image processing apparatusand an image processing method corresponding to the above imageprocessing apparatus also constitute another aspect of the embodiments.

Further features of the disclosure will become apparent from thefollowing description of exemplary embodiments with reference to theattached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram showing a configuration of an image processingapparatus according to one embodiment of the disclosure.

FIG. 2 illustrates a relationship between a pixel luminance andpolarization information.

FIG. 3 is a flowchart showing image processing according to first andsecond embodiments.

FIGS. 4A to 4C illustrate a captured image according to the firstembodiment.

FIGS. 5A and 5B illustrate degree of polarization histograms accordingto the first embodiment.

FIG. 6 illustrates a threshold area according to the first embodiment.

FIGS. 7A to 7D illustrate combined images according to the firstembodiment.

FIG. 8 illustrates extracted areas in the first embodiment and acomparative example.

FIGS. 9A to 9D illustrate a configuration of a polarizing elementaccording to the first embodiment.

FIG. 10 illustrates a configuration of an image processing systemaccording to a third embodiment of the disclosure.

FIG. 11 illustrates a configuration of an image pickup apparatusaccording to the third embodiment.

FIG. 12 is a block diagram showing a configuration of an imageprocessing apparatus according to fourth to sixth embodiments.

FIG. 13 is a flowchart showing image processing according to the fourthembodiment.

FIG. 14 is a flowchart showing image processing according to the fifthembodiment.

FIG. 15 is a flowchart showing image processing according to the sixthembodiment.

FIG. 16 illustrates luminance I and polarization information A, B, and θagainst a polarization angle.

FIG. 17 illustrates an image pickup apparatus that includes the imageprocessing apparatus according to one of the fourth to sixthembodiments.

FIG. 18 is a flowchart showing image processing according to acomparative example.

DESCRIPTION OF THE EMBODIMENTS

Referring now to the accompanying drawings, a description will be givenof embodiments according to the disclosure. FIG. 1 illustrates aconfiguration of an image processing apparatus 101 according to oneembodiment of the disclosure. The image processing apparatus 101includes a polarization calculator (acquisition unit) 11, a polarizationcorrector (correction unit) 12, and an image combiner (generation unit)13.

The polarization calculator 11 acquires an RGB image (referred to as acaptured image hereinafter) as an input image generated by imaging lightof a plurality of colors (referred to as R, G, and B in the examples) indifferent polarized states using an image pickup apparatus describedlater in a third embodiment. At this time, one or more images areacquired as captured images. The polarization calculator 11 calculatesfirst polarization information for each of R, G, and B in each pixelfrom luminance information of the acquired captured image.

The polarization corrector 12 calculates second polarization informationusing information of a specific color among R, G, and B in a specificarea of the captured image, and generates corrected polarizationinformation made by replacing the first polarization informationcalculated by the polarization calculator 11 with the calculated secondpolarization information. The polarization combiner 13 generates(combines) a combined image as an output image using at least thecorrected polarization information generated by the polarizationcorrector 12.

A detailed description will now be given of generations of the firstpolarization information acquired by the polarization calculator 11 andthe corrected polarization information generated by the polarizationcorrector 12.

When an image is acquired by imaging through an element (such as apolarizing plate) that transmits linearly polarized light having aspecific polarization direction, there is a relationship represented bythe following expression (1) between an angle α of the transmittinglinearly polarized light (referred to as a polarization anglehereinafter) and a luminance value (luminance information) I_(j)(α) ineach pixel in the acquired image:I _(j)(α)=A _(j) cos²(α−θ_(j))+B _(j)  (1)

FIG. 2 illuminates the luminance value I_(j)(α) relative to thepolarization angle α of a certain color channel in a certain pixel,where j represents one of the colors R, G, and B and, for example, I_(R)represents the luminance value in a certain pixel R. A_(j) represents aluminance component that changes according to the polarization angle α(referred to as a polarized light component hereinafter), and B_(j)represents a luminance component that does not change according to thepolarization angle α (referred to as a nonpolarized light componenthereinafter). θ_(j) represents a polarization angle that maximizes theluminance value I(α). In other words, θ_(j) is polarization anglecorresponding to maximum luminance value. θ_(j) is referred to as anangle component hereinafter. A_(j), B_(j) and θj are constantscalculated for each pixel and for each color.

Changing the polarization angle α to three or more different angles andacquiring an image can provide A_(j), B_(j), and θ_(j) in the expression(1). Thereby, the relationship between the polarization angle α and theluminance value I_(j)(α) can be acquired for each color channel and foreach pixel.

The angle component θ_(j) usually does not change depending on thecolor. This is because the polarized light component A_(j) is mainlyderived from the specular reflection component. Since the specularreflection component follows the Fresnel's law, a reflectance ofs-polarized light is generally higher than that of p-polarized light.Thus, the direction of the s-polarized light and the direction of theangle component θ_(j) coincide with each other. The direction of thes-polarized light is determined by the incident plane and the emissionplane, and does not depend on the wavelength (color).

Due to some errors, the angle component θ_(j) may be differentlycalculated depending on the color. In particular, when the polarizationangle α is controlled with a polarizing element including a variableretardation plate disclosed in JP 2017-191986, it is affected by theangular characteristic of the variable retardation plate, the wavelengthdependence of the retardation, and the like, the influence of thewavelength (color) becomes remarkable.

FIG. 9A illustrates a configuration of a polarizing element 200disclosed in JP 2017-191986, and FIGS. 9B to 9D illustrate a quarterwaveplate 21, a variable retardation plate 22 including a liquidcrystal, and a polarizing plate 23, which are elements of the polarizingelement 200, respectively. Arrows in FIGS. 9B to 9D point to any of afast axis, a slow axis, and a transmission axis. A z-axis indicates anoptical axis direction during imaging, and the x-axis and the y-axisindicate axes in a plane orthogonal to the optical axis (z-axis) andorthogonal to each other. Angles formed by the axial directions of thequarter waveplate 21, the variable retardation plate 22, and thepolarizing plate 23 relative to the x-axis are 90°, 45°, and 90°,respectively.

The quarter waveplate 21 gives a relative retardation (phase difference)of π/2 (rad) between the polarized light components orthogonal to eachother in the incident light. The relative retardation of π/2 given bythe quarter waveplate 21 is invariant (fixed). The variable retardationplate 22 is an element using a liquid crystal, and gives a relativeretardation that is variable according to the voltage applied betweenthe orthogonal polarized light components. The polarizing plate 23transmits a polarized light component in the transmission axisdirection.

When RGB light transmits through the thus configured polarizing element200, the peak position of the polarization angle componentcos²(α_(c)−θ_(j)) at the specific polarization angle α_(c) shifts amongR, G and B, and the combined image generated with the polarization anglecomponent cos²(α_(c)−θ_(j)) at the specific polarization angle α_(c) hasunnatural coloring (unnecessary color) that would not occur in a normalimage. Since this coloring is caused by the shift of the polarizationangle component cos²(α_(c)−θ_(j)) peak position among R, G, and B,coloring changes depending on the value of the specific polarizationangle α_(c) used to generate the combined image.

In the embodiment, in order to reduce the coloring that varies dependingon the value of the specific polarization angle α_(c), the polarizationcorrector 12 generates (information of) the corrected angle component asthe corrected polarization information, and the polarization combiner 13generates the combined image using the corrected angle component.

The corrected angle component is generated by replacing an anglecomponent (first polarization information) for each color channelcalculated for each pixel by the polarization calculator 11 in aspecific area in the captured image, with a separately calculated anglecomponent (second polarization information). The separately calculatedangle component is an angle component calculated using information of aspecific color among R, G, and B. For example, when G is selected as aspecific color, information on θ_(G) of all pixels is calculated usingthe luminance value of G and the expression (1) in the specific area,and θ_(R), θ_(G), and θ_(B) calculated by the polarization calculator 11are replaced with θ_(G). This processing makes θ_(R), θ_(B) and θ_(G)equal to each other, and can reduce coloring that varies depending onthe value of α_(c) described above.

When the polarization calculator 11 has calculate θ_(R), θ_(G), andθ_(B) in advance using the respective luminance values of R, G, and B,the processing of replacing OR and OB for each pixel with θ_(G) in thespecific area. On the other hand, when the luminance distribution of Rand G is also taken into consideration in calculating θ_(G), thepolarization corrector 12 recalculates θ_(G)″ based on the luminancevalue of G, and replaces θ_(R), θ_(G), and θ_(B) with θ_(G)″.

The specific color used to calculate the corrected angle component canbe arbitrarily selected from R, G, and B. For example, when a capturedimage acquired with a Bayer array image sensor is used, G having alarger number of pixels than that of another color can be selected asthe specific color.

A specific color may be selected according to the polarizationinformation and the color information of the object image in thespecific area. For example, the accuracy of the polarization informationtends to be higher, as the degree of polarization (DOP) defined by thefollowing expression (2) becomes higher:DOP_(j)=(I _(jmax) −I _(jmin))/(I _(jmax) +I _(jmin))=A _(j)/(A _(j)+2B_(j))  (2)

I_(jmax) and I_(jmin) in the expression (2) are a maximum value and aminimum value of the luminance value I_(j)(α) when the polarizationangle α is changed in any of the color channels of R, G, and B,respectively.

A high corrected angle component can be obtained by calculating the DOPwithin the specific area based on the polarization informationcalculated by the expression (2) and the polarization calculator 11 andby selecting the color with the highest DOP among R, G, and B as thespecific color.

Another method compares the luminance values of R, G, and B of theobject (the area in which the object is reflected) in the specific area,and selects the color having the lowest luminance value (such as B for ayellow object) as the specified color. Since the DOP increases as thediffusion component becomes lower, the luminance value for each of R, G,and B of the object in the captured image tends to be inverselyproportional to the DOP for each of R, G and B. Thus, even this methodcan indirectly select a color having a high DOP as the specific color.

The method of selecting any one of R, G, and B as the specific color hasbeen described, but two colors (such as B and G, R and B, etc.) may beselected. Even in that case, as described above, two colors may beselected based on the DOP, the color of the object, the number ofpixels, and the like. The above description generates the correctedangle component using the angle component of the specific color byreplacing θ_(B), θ_(G), and θ_(R), but the corrected angle component maybe generated by replacing α_(c)−θ_(j) and cos²(α_(c)−θ_(j)).

This replacement processing of the angle component in the specific areacan reduce a calculation amount in comparison with the processing of theentire image. Thus, the specific area may be selected from an area thatrequires the replacement processing of the angle component, that is, anarea in which coloring is likely to occur. For example, an area may beselected in which the DOP is larger than a predetermined value(threshold) or an area in which a difference in the angle component θfor each of R, G, and B is larger than the threshold.

The specific area is not limited to one, and a plurality of specificareas may be selected from the entire image. In selecting multiplespecific areas, an arbitrary classification method or an arbitrarymethod may be selected or separated, such as specifying a specific areafor each specific color used to calculate the corrected angle component,separating a specific area for each object, and separating a specificarea for the background and the main object. The entire image may bedivided into a plurality of areas in advance according to the object,color, DOP, etc., and the specific area may be selected from theplurality of divided areas.

First Embodiment

A flowchart in FIG. 3 shows processing (image processing method)according to a first embodiment executed by the image processingapparatus 101 illustrated in FIG. 1 . The image processing apparatus 101is configured by a computer and executes this processing according to acomputer program.

In the step S101, the image processing apparatus 101 reads polarizedimage data as a captured image generated by imaging each of R, G, and Bin different polarization states. The polarized image data to be read isa colored image having luminance information for each of R, G, and B,and is a RAW image that has received no compression processing or gammaprocessing. In the embodiment, an image is captured with the polarizingelement 200 shown in FIGS. 9A to 9D by an image pickup apparatusprovided with an imaging sensor in a Bayer array. The polarizing element200 can modulate the polarization angle α, which is the polarizationdirection of the transmitting light, by changing the in-planeretardation (phase difference) Δ imparted by the variable retardationplate 21 to the transmitting light. A relationship between thepolarization angle α (deg) of the transmitting light and the in-planeretardation Δ (nm) is expressed by the following expression (3):α=Θ−180·Δ/λ  (3)

In the expression (3), Θ is a transmission axis angle (deg) of thepolarizing plate 23 disposed in the polarizing element 200, and λ is awavelength (nm) of the transmitting light. From the expression (3), thecaptured image obtained by imaging through the polarizing element 200using the variable retardation plate 21 has a different polarizationangle α depending on the wavelength λ.

FIGS. 4A to 4C illustrate an example of three captured images (polarizedimage data) acquired in this embodiment. The captured images in FIGS. 4Ato 4C are obtained by imaging the polarizing plate 23 at thetransmission axis angle Θ of 90° and in-plane retardations Δ of 10 nm,133 nm, and 265 nm, respectively. Table 1 shows the polarization anglesα (deg) of the R, G, and B color channels at this time.

TABLE 1 Δ (nm) 10.0 133.0 265 B 86.2 39.3 −11.5 G 86.6 45.0 0.0 R 87.050.3 10.5 α (deg)

Table 1 sets the wavelengths used to calculate the transmission anglesof the R, G, and B color channels to 600 nm, 530 nm, and 470 nm,respectively, and selects the wavelength that maximizes the transmissionlight amount of the color filters disposed in the pixels of R, G, and B.

Next, in the step S102, the image processing apparatus 101 (polarizationcalculator 11) plots the luminance value of each pixel relative to thepolarization angle α from the luminance value data of the three capturedimages shown in FIGS. 4A to 4C, and calculates the polarizationinformation A_(j), B_(j), and θ_(j) of each pixel. More specifically,since any of R, G, and B luminance values is acquired for each pixel ofthe captured image, the value of the transmission angle α in Table 1 isselected according to R, G, or B of the pixel, and the polarizationinformation A_(j), B_(j), θ_(j) of any of R, G, and B is calculatedaccording to the pixel. Then, demosaic processing to the captured imagefinds the polarization information A_(j), B_(j), and θ_(j) for R, G, andB of all the pixels. A general method may be used for the demosaicprocessing. In consideration of the subsequent processing, thisembodiment performs the demosaic processing using the luminance value ofeach color for each color channel.

This embodiment acquires the polarization information, and then thepolarization information of R, G, and B of all the pixels using thedemosaic processing, but may perform the demosaic processing for theluminance value of the captured image, and then calculate thepolarization information for each pixel and each color. Nevertheless, amethod of calculating the polarization information and then performingthe demosaic processing is advantageous due to a smaller calculationamount.

Object alignment processing may be performed as necessary before thepolarization information is calculated from the three captured images.

After the polarization information A_(j), B_(j), and θ_(j) for R, G, andB of all the pixels is calculated in this way, in the step S103, theimage processing apparatus 101 (polarization corrector 12) selects aspecific area from the entire area of the captured image. As describedabove, the difference of θ_(j) caused by the DOP or RGB is compared withthe threshold, and an area in which the DOP is larger than the thresholdor an area in which the difference of θ_(j) is larger than the thresholdis selected as the specific area. In this embodiment, an area having theDOP larger than the threshold is selected as the specific area. Inaddition, in this embodiment, the threshold is determined with ahistogram of the DOP.

FIGS. 5A and 5B illustrate histograms of the degree of polarization(DOP) used to determine the threshold. FIG. 5A illustrates a histogramwith a DOP of 0 to 1, and FIG. 5B illustrates a histogram with a DOP of0.05 to 1. From FIGS. 5A and 5B, it can be estimated that an area havinga DOP less than 0.1 is a background image, and an area having a DOParound 0.6 is an area having a DOP peak in the object. This embodimentdetermines the DOP of 0.45, which is around 0.6, as a threshold andselects an area having a DOP of 0.45 or higher as the specific area soas to exclude the background area and to sufficiently correct thecolored area of the object.

FIG. 6 illustrates a specific area selected by the above method in thecaptured image. In FIG. 6 , an area having a DOP of 0.45 or higher isexpressed by white, and an area having a DOP of less than 0.45 isexpressed by black. From the comparison between FIGS. 6 and 4A to 4C, itis understood that selecting the area having a DOP of 0.45 or higher asthe specific area can select the area in which the luminance valuevaries depending on the polarization state.

Setting the threshold for the DOP is not limited to the method using thehistogram of the DOP described above, and an arbitrary threshold may beused. For example, although it depends on the captured image, it may beusually set to 0.1 or higher, or 0.3 or higher. The specific area inwhich the difference of θ_(j) is larger than the threshold can beselected in the same manner as that when the DOP is used.

Next, in the step S104, the image processing apparatus 101 (polarizationcorrector 12) selects a specific color used to generate the correctedpolarization information. Assuming that the object is almost achromatic,this embodiment selects G that is assigned to many pixels as a specificcolor.

In selecting the specific color using the hue of the captured image, thehue of the captured image may be determined with the image thatminimizes the absolute value of the retardation of the variableretardation plate among the captured images (the captured image shown inFIG. 4A in this embodiment). This is because the wavelength dispersionof the polarization angle α increases as the absolute value of theretardation of the variable retardation plate increases, andconsequently the captured image comes to include coloring due to thewavelength dispersion of the polarization angle α_(j).

In the step S105, the polarization corrector 12 that has selected thespecific color generates (calculates) the corrected polarizationinformation using the information of G, which is the specific color. Incalculating the polarization information in the step S102, thisembodiment calculates the polarization information of R, G, and B ofeach pixel using the luminance values of R, G, and B, respectively.Therefore, this embodiment generates the corrected polarizationinformation by replacing θ_(R) and θ_(B) of each pixel in the specificarea with θ_(G).

Next, in the step S106, the image processing apparatus 101 (imagecombiner 13) generates a combined image using the corrected polarizationinformation and the polarization information. This embodiment calculatesthe luminance value I_(j)′ of each pixel in the combined image using thefollowing expression (4).I _(j)′(α_(c))=k _(l) A _(j) cos² [k ₂(α_(c)−θ_(j)′)]+k ₃ B _(j)  (4)

In the expression (4), k₁ to k₃ are arbitrary coefficients, and α_(c) isan arbitrary polarization angle. θ_(j)′ is the corrected polarizationinformation (corrected angle component) generated in the step S105, andA_(j) and B_(j) are the polarized light component and the nonpolarizedlight component calculated in the step S102, respectively.

The coefficients k₁ to k₃ and the polarization angle α_(c), use constantvalues regardless of R, G, and B and pixels. Setting these coefficientsand the polarization angle to constant values regardless of the color orpixel can unify the texture in the image given to the combined image bythe polarization information. These coefficients do not necessarily haveto be equal in the entire image, and may be set for each area in theimage, for each object, or for each area where the texture is desired tobe unified.

This embodiment sets k₁=k₂=k₃=1 over the entire image, and generatesfour combined images at α_(c)=0°, 45°, 90°, and 135°. FIGS. 7A to 7Dshow an example of combined images generated under this condition. Thecombined images actually generated are colored images, but are shown asmonochromatic images in FIGS. 7A to 7D. From these figures, it isunderstood that generating the combined images by changing thepolarization angle α_(c) can produce a combined image having a differentgloss position.

Finally, in the step S107, the image processing apparatus 101 outputsthe generated combined image and ends this processing.

This embodiment sets the in-plane retardation Δ to a constant value forR, G, and B, but Δ may be different for R, G, and B. If Δ is differentfor R, G, and B, the polarization angle α may be calculated using anappropriate Δ for each color channel.

Comparative Example

The effect of using the corrected angle component in the firstembodiment will be compared with a comparative example that does not usethe corrected angle component. This comparative example uses the anglecomponent θ_(j) instead of the corrected angle component θ_(j)′ ingenerating the combined image, and generates four combined images(α_(c)=0°, 45°, 90°, 135°) similar to the first embodiment using thesame method as that of the first embodiment other than the above.

Table 2 shows a calculation result of L*a*b* representative values ineach combined image so as to confirm coloring depending on thepolarization angle α_(c) in the four combined images generated in thecomparative example and the four combined images generated in the firstembodiment (simply referred to as embodiment in the table). The L*a*b*representative values in each combined image are made by extracting thesame partial area (the area in the white frame on the lens cap of theinterchangeable lens shown in FIG. 8 ) in the four combined images, andby calculating the average value of L*a*b* in the same partial area.Table 2 also shows the average and dispersion of the L*a*b*representative values in each combined image.

TABLE 2 COMPARATIVE EMBODIMENT EXAMPLE α_(c) L* a* b* L* a* b* 0 60.61.395 −3.207 60.5 1.071 −3.694 45 48.4 1.275 −3.457 48.7 3.337 −1.544 9022.4 −0.120 −3.212 22.4 1.161 −2.802 135 43.5 0.874 −2.890 43.3 −2.163−5.475 AVERAGE 43.7 0.856 −3.191 43.7 0.852 −3.379 DISPERSION 189.80.355 0.040 189.4 3.852 2.048

In the representative values of L*a*b* in the first embodiment and thecomparative example in Table 2, the value of L* representing thelightness is almost the same in the first embodiment and the comparativeexample, whereas the values of a* and b* representing the chromaticitydiffer between the first embodiment and the comparative example, and thevariation in the comparative example is larger than that in the firstembodiment. When the dispersions of a* and b* are compared, thedispersion of the comparative example is larger than that of the firstembodiment. From this fact, it can be seen that a change in tint betweenthe combined images is smaller in the first embodiment than in thecomparative example, and using the corrected angle component can reducecoloring that varies depending on the value of α_(c).

Second Embodiment

A second embodiment according to the disclosure will be described. Thesecond embodiment is different from the first embodiment in that thesecond embodiment calculates, as the corrected polarization information,(information of) a corrected polarized light component A_(j)′ and acorrected nonpolarized light component B_(j)′ as well as the correctedangle component θ_(j)′, and generates a combined image using θ_(j)′,A_(j)′, and B_(j)′.

The second embodiment executes the steps S101 to S104 in the imageprocessing shown in FIG. 3 , similar to the first embodiment. Thecalculation of the corrected angle component θ_(j)′ in the step S105 isthe same as that in the first embodiment.

In the second embodiment, in the step S105, the image processingapparatus 101 (polarization corrector 12) further calculates thecorrected polarized light component A_(j)′ and the correctednonpolarized light component B_(j)′ using the corrected angle componentθ_(j)′. The corrected polarized light component A_(j)′ and the correctednonpolarized light component B_(j)′ are generated by replacing apolarized light component and a nonpolarized light component for eachcolor calculated for each pixel by the polarization calculator 11 in thespecific area in the captured image, with a polarized light componentand a nonpolarized light component calculated separately. The separatelycalculated polarized light component and nonpolarized light componentare calculated from the expression (1) using the corrected anglecomponent θ_(j)′ and the luminance value data of the three capturedimages.

More specifically, the corrected angle component θ_(j)′ in which θ_(R)and θ_(B) of each pixel in the specific area are replaced with θ_(G) isset to θ_(j) in the expression (1), and the polarized light componentA_(j) and the nonpolarized light component B_(j) for each pixel areagain obtained from the luminance value data of the three capturedimages. Then, the corrected polarized light component A_(j)′ and thecorrected nonpolarized light component B_(j)′ are made by replacingpolarized light components A_(R) and A_(G) and nonpolarized lightcomponents B_(R) and B_(G) calculated by the polarization calculator 11with the polarized light component A_(j) and the nonpolarized lightcomponent B_(j). This processing can correct the error contained in thepolarized light component and the nonpolarized light component.

In the step S1076, the image processing apparatus 101 (image combiner13) generates a combined image using θ_(j)′, A_(j)′, and B_(j)′.

While the first embodiment generates a combined image using thecorrected angle component and the polarization information (polarizedlight component and nonpolarized light component) calculated in the stepS102, the second embodiment generates a combined image using thecorrected polarization information (corrected polarized light component,corrected nonpolarized light component, and corrected angle component).This configuration can reduce coloring caused by the error between thepolarized light component and the nonpolarized light component.

In the first and second embodiments, the polarization calculator 11performs the demosaic processing and calculates the polarizationinformation of all pixels and all color channels, but the timing of thedemosaic processing can be changed as appropriate. For example, afterthe luminance value of each pixel is calculated using the expression (4)in generating the combined image, the demosaic processing may beperformed. At that time, generate the polarization information of eachpixel in the specific area from the information of peripheral pixels incalculating the corrected polarization information is generated.

The polarizing element used for imaging for a captured image is notlimited to the polarizing element 200 shown in FIGS. 9A to 9D. Forexample, various polarizing elements can be used, such as a polarizingplate, a C-PL filter that combines a polarizing plate and a quarterwaveplate, and a polarizing sensor in which a polarizing element in adifferent direction is incorporated in each pixel on an imaging sensor.

Generating the combined image is not limited to the method using theexpression (4) and, for example, a combined image may be generated bycalculating the luminance value I_(j)′ of each pixel using a pluralityof polarization angles α_(c1) to α_(cn) as shown in the followingexpression (5):I _(j) ′=Σk _(ln) A _(j) cos² [k _(2n)(α_(cn)−θ_(j))]+k ₃ B _(j)  (5)

In the expression (5), k_(1n), k_(2n), and k₃ are arbitrarycoefficients, and α_(cn) is an arbitrary polarization angle. Thecoefficients and polarization angle use constant values regardless of R,G, and B and the pixel, similar to the expression (3). The combinedimage may be generated with part of the expression (4) or (5).

General image processing such as noise reduction processing, contrastcorrection processing, and white balance correction processing may beperformed for the captured image, the combined image, the polarizationinformation, and the corrected polarization information.

The captured image does not necessarily have to be a raw image, and acompressed image such as a JPEG image can also be used. When a capturedimage in which the luminance value is gamma-corrected is used, a reversecorrection may be performed before the polarization information iscalculated to return the luminance value to a linear shape.

Third Embodiment

FIG. 10 illustrates the configuration of the image processing system 100including the image processing apparatus 101 described above.

The image processing apparatus 101 is a computer device equipped withimage processing software (an image processing program as a computerprogram) 106 and executes the image processing described in the first orsecond embodiment in accordance with the image processing software 106.

The image pickup apparatus 102 is a device such as a camera, atelescope, an endoscope, and a scanner, which acquires an image throughimaging. FIG. 11 illustrates a lens interchangeable type camera as anillustrative image pickup apparatus 102. The interchangeable lens cameraincludes an interchangeable lens 121, an adapter 122, and a digitalcamera body 123. The adapter 122 is provided inside the polarizingelement 200 illustrated in FIG. 9A.

The digital camera body 123 includes an image sensor (image pickupelement) 123 a such as a CMOS sensor, and can image light that hastransmitted through the interchangeable lens 121 and the adapter 122.

The polarizing element (adapter 122) is provided between theinterchangeable lens 121 and the digital camera body 123 in FIG. 11 ,but may be provided on the object side of the interchangeable lens 121.The adapter 122 may not be provided, and a polarizing element may beprovided integrally with or just before the image sensor in the digitalcamera body 123.

A storage medium 103 shown in FIG. 10 , such as a semiconductor memory,a hard disk drive, and a server on a network, stores images acquired byimaging.

The image processing apparatus 101 acquires a captured image as an inputimage from the image pickup apparatus 102 or the storage medium 103 by awired or wireless communication with it or by reading through anattachment. Then, an output image is generated by the image processingdescribed in the first or second embodiment, and output to at least oneof the output device 105, the image pickup apparatus 102, and thestorage medium 103. The output image can be stored in the internalmemory built in the image processing apparatus 101. The output device105 includes, for example, a printer.

A display device 104 is also connected with the image processingapparatus 101. Thus, the user can perform image processing work andevaluate the generated output image through the display device 104. Inaddition to the image processing described in the first or secondembodiment, the image processing apparatus 101 may perform other imageprocessing such as development processing and image restorationprocessing, if necessary.

In this embodiment, the image processing apparatus 101 is separate fromthe image pickup apparatus 102, but it may be built in the image pickupapparatus 102.

Fourth Embodiment

FIG. 12 illustrates a configuration of an image processing apparatus1200 according to fourth to sixth embodiments of the disclosure. Theimage processing apparatus 1200 receives a plurality of colored imagescaptured with different polarized states as input images. It acquirespolarization information from these input images, and generates acolored image (referred to as a combined image hereinafter) as an outputimage different from the input image based on the acquired polarizationinformation. The image processing apparatus 1200 includes (a processoror circuits that serves as) an image acquirer 1201, a polarizationinformation acquirer 1202, and an image combiner 1203.

The image acquirer 1201 acquires a plurality of input images withpolarization states different from each other, which have been generatedthrough imaging by an image pickup apparatus that includes at least apolarizing element having a variable retardation plate and a polarizingplate, and an image sensor configured to acquire a colored image. Thepolarization information acquirer (acquisition unit and setting unit)102 sets a representative wavelength for each color channel of the inputimage, and acquires the polarization information using therepresentative wavelength and the plurality of input images. The imagecombiner (generating unit) 103 generates an output image using theacquired polarization information.

Next follows a description of acquiring the polarization information bythe polarization information acquirer 1202. When an image is capturedthrough an element (such as a polarizing plate) that transmits specificlinearly polarized light, an angle φ of the transmitting linearlypolarized light (hereinafter referred to as the polarization angle) andan luminance I (φ) in each pixel of the captured image has the followingrelationship:I(φ)=A cos²(φ−θ)+B  (6)

When images are captured while the polarization state is modulated andthe polarization angle φ is changed to three or more different states,the polarization information A, B, and θ in the expression (6) can befound for each pixel and the luminance I(φ) against the polarizationangle φ can be found. The polarization information may be calculated byfitting luminance values of four or more captured images using theexpression (6). At that time, for example, the least squares method orthe like can be used.

FIG. 16 illustrates the luminance I(φ) against the polarization angle φand the polarization information A, B, and θ. Here, A is a luminancecomponent that depends on the polarization angle φ, B is a luminancecomponent that does not depend on the polarization angle φ, and θ is aphase that maximizes the luminance I(φ) (referred to as a phasecomponent hereinafter).

When the means for modulating the polarization state uses a polarizingelement including a waveplate, a variable retardation plate, and apolarizing plate as disclosed in the first embodiment in JP 2016-145924,the polarization angle φ and the wavelength λ (nm) of the incident lighthas the following relationship:φ(λ)=φp−πδ/λ  (7)

Here, δ is a phase difference (retardation) amount (nm) given by thevariable retardation plate, and φp (rad) is a transmission polarizationangle of the polarizing plate in the polarizing element. As shown by theexpression (7), the polarization angle φ (rad) varies depending on thewavelength λ.

Light having a certain wavelength width for each color channel usuallyenters each pixel in the image sensor, and the luminance signal I isacquired through the photoelectric conversion. Here, the acquiredluminance signal I is the total luminance of light having a certainwavelength width, and cannot be separated for each different wavelength.Hence, in order to define the polarization angle for each color channeland to acquire the polarization information for the plurality of inputimages, the representative wavelength λ is set for each color channel,and the polarization information A, B, and θ is calculated for eachcolor channel by using the polarization angle φ(λ) calculated from therepresentative wavelength.

If the representative wavelength λ for each color channel is notproperly set, the calculated polarization information A, B, and θincludes large calculation errors and consequently the combined imagecreated with the polarization information contains unnatural coloring.JP 2017-191986 discloses a method of setting the representativewavelength which is a method of setting a transmittance peak wavelengthto the representative wavelength from the transmission wavelengthdistribution of a color filter used for an image sensor. However, whenthe intensity wavelength distribution of the illumination light and thetransmission wavelength distribution of the color filter aresignificantly different, coloring cannot be fully reduced and unusuallyappears in the combined image. In particular, the influence of coloringbecomes remarkable under illumination in which the spectrum of theillumination light has a steep peak against the wavelength, such as theLED illumination or fluorescent lamp illumination.

Accordingly, this embodiment pays attention to a difference in θ betweencolor channels in each pixel and sets the representative wavelength ofeach color channel so as to reduce the difference. This method canreduce coloring in the combined image. A description will now be givenof a method for setting the representative wavelength.

Of the polarization information A, B, and θ, the phase component θ isinformation that does not depend on the color channel. This is becausethe reflected light on the surface of the object follows the Fresnel'slaw, so that the reflectance of the s-polarized light is larger thanthat of the p-polarized light except for some conditions and the phasecomponent θ coincides with the direction of the s-polarized light. Thus,the phase component θ is determined by the surface normal of the objectand does not depend on the wavelength (or the color channel). However,if the representative wavelength of each color channel is improperlyselected and the calculation error included in the polarizationinformation becomes large, the difference in the phase component θbetween the color channels becomes large. That is, there is a positivecorrelation between the difference of the phase component θ and thecalculation error included in the polarization information. Selectingthe representative wavelength of each color channel so that thedifference in θ for each color channel becomes as small as possible canreduce the error of the polarization information and suppress coloringin the combined image.

The polarization information acquirer 1202 serves to evaluate thedifference in the phase component θ between the color channels when thepolarization information is calculated by changing the representativewavelength, and finally sets the representative wavelength λ of eachcolor channel that makes the difference in the phase component θ equalto or less than the threshold or minimizes it. Thereby, therepresentative wavelength of each color channel can be optimized.

In evaluating the difference in the phase component θ for each colorchannel, the total value of the differences in all the pixels may beevaluated, but paying attention to pixels having a high DOP and reducingthe difference in the pixels can further improve the coloring reducingeffect in the combined image. Therefore, the difference in the phasecomponent θ for each color channel may be evaluated based on the totalvalue of the differences in the pixels having a DOP equal to or largerthan a predetermined value among all the pixels or the total value ofthe differences weighted according to the DOP of each pixel.

In evaluating the difference in the phase component θ, one method mayevaluate the difference in the phase component θ between one colorchannel as a reference and at least another color channel. The method ofselecting the reference color channel is not particularly limited but,for example, one of the plurality of color channels, which has thelargest number of pixels in the image sensor used for imaging may beselected as the reference. Alternatively, a color channel with thehighest DOP may be selected as a reference or a color channel may beselected as a reference which minimizes the difference between themaximum and minimum wavelengths of the light incident on each colorchannel and the minimum and maximum values of φ(λ) calculated from theexpression (7).

The representative wavelength of each color channel may be set to adifferent value for each pixel or may be set to the same value in theentire image or a specific area of the image. This method can reduce thenoise effect in the captured image.

There is no particular limitation on the method of selecting a specificarea, and a general image area dividing method can be used. Since therepresentative wavelength depends on the refractive index of the object,the wavelength distribution of the illumination light, and the like, itmay be divided into areas where they are estimated to be equal to orclose to each other. More specifically, the image may be divided intoareas according to the type and color of the object, or when a pluralityof types of illumination are used, the image may be divided for eacharea of the illumination light.

The optimized representative wavelength can be used for the whitebalance correction for the combined image. Conventionally, the whitebalance correction includes auto white balance control thatautomatically detects an area estimated to be white from an imagegenerated by imaging and obtains a correction coefficient for the area.This control divides the image into a plurality of blocks, andcalculates the color evaluation value for each block. A block with acalculated color evaluation value included in the preset white detectingrange is determined to be white, and the correction value is calculated.

This aspect of the embodiments serves to adjust the white detectingrange according to the optimized representative wavelength. Thisfunction can reduce erroneous determinations, such as detecting an areathat is not originally white or failing to detect a white area, and canimprove the accuracy of the white balance correction.

A flowchart in FIG. 13 shows image processing (method) executed by animage processing apparatus 1200 (including a processor or circuit thatserves as the image acquirer 1201, the polarization information acquirer1202, and the image combiner 1203) configured by a computer in thefourth embodiment of the disclosure in accordance with a computerprogram. In FIG. 13 , “S” stands for the step.

In S1300, as described above, the image processing apparatus 1200acquires a plurality of input images generated by an image pickupapparatus configured to capture a plurality of polarized light beamshaving different polarization states from each other and provided withat least a polarizing element having a variable retardation plate and apolarizing plate, and an image sensor configured to acquire a coloredimage.

In S1301, the image processing apparatus 1200 sets the representativewavelength for each color channel to a preset initial value, andcalculates the polarization angle φ (λ) for each color channel for eachinput image. The initial value does not have to be a single value, andthe image processing apparatus 1200 may automatically select it fromamong a plurality of initial values, or the user may manually select theinitial value. The image processing apparatus 1200 automatically selectsit using correction information calculated by the conventional whitebalance correction on the input image that is captured in the statewhere the phase difference (retardation) given by the variableretardation plate is the least among the plurality of input images.

Next, in S1302, the image processing apparatus 1200 acquires thepolarization information A, B, θ for each color channel in each pixelusing the polarization angle φ (λ) calculated in S1301, each inputimage, and the expression (6).

Next, in S1303, the image processing apparatus 1200 compares the phasecomponents θ calculated for each color channel with each other, anddetermines whether or not the difference between them is equal to orless than a predetermined threshold. At this time, it sets one of theplurality of color channels to a reference, and evaluates the differencefrom the reference color channel. The threshold may be 20° or less, 10°or less, or 5° or less.

When the difference in the angle component θ is larger than thethreshold, the image processing apparatus 1200 returns to S1301, resetsthe representative wavelength, and repeats the subsequent processing.Thereby, the representative wavelength is sequentially changed from theinitial value. If the difference in the angle component θ is equal to orless than the threshold value, the flow proceeds to S1304, and generatesa combined image different from the input image using A, B, and θacquired in S1302.

In S1303, even if the difference is equal to or less than the threshold,if the number of repetitions of S1301 to S1303 is less than thespecified number of times, the representative wavelength may be resetand the processing may be repeated. In this case, the image processingapparatus 1200 may reset the representative wavelength and repeats theprocessing until the number of repetitions of S1301 to S1303 reaches thepredetermined number, sets the representative wavelength that minimizesthe difference to the final representative wavelength, and generates acombined image using the polarization information at the representativewavelength. S1301 to S1303 may be repeated until the number ofrepetitions reaches the specified number of times without setting thethreshold, and the combined image may be created with the polarizationinformation at the representative wavelength where the difference isminimized.

Comparative Example

Referring now to a flowchart in FIG. 18 , a description will be given ofa conventional image processing apparatus according to a comparativeexample of the fourth embodiment, which sets, to a representativewavelength, a transmittance peak wavelength of the transmittancedistribution of the color filter in the image sensor.

In S1800, the conventional image processing apparatus shown in JP2017-191989, similar to the image processing apparatus 1200 of thefourth embodiment, acquires a plurality of input images having differentpolarization states generated by an image pickup apparatus that includesa polarizing element having a variable retardation plate and apolarizing plate, and an image sensor configured to acquire a coloredimage.

In S1801, the image processing apparatus sets the transmittance peakwavelength of the color filter for each color corresponding to eachcolor channel to the representative wavelength for each color channel.S1802 and S1803 are the same as S1302 and S1304 in the fourthembodiment.

This comparative example does not execute the processing of determiningthe difference in the phase component θ for each color channel, which isperformed in S1303 in the fourth embodiment. If the representativewavelength set in S1801 is improper, the combined image created in S1803will be unnaturally colored. Thus, in S1804, the image processingapparatus performs color correction processing so that the referenceimage and the combined image have the same color information in order toeliminate coloring of the combined image created in S1803. Here, thereference image is set to one of input images which has the least phasedifference (retardation) given by the variable retardation plate duringimaging. When the reference image selected in S1804 has a colored areadifferent from the color of the object due to the influence of shading,gloss, etc., the color correction processing in S1804 cannot correctlycorrect coloring of the combined image.

Fifth Embodiment

A flowchart in FIG. 14 shows image processing executed by an imageprocessing apparatus 1200 according to a fifth embodiment of thedisclosure. In this embodiment, a plurality of captured images acquiredin S1400 similar to S1300 of the fourth embodiment are divided intoareas (blocks) in S1401.

Then, the image processing apparatus 1200 sets an initial value of arepresentative wavelength in S1402 for each divided area correspondingto each other in the plurality of captured images, similar to S1301 ofthe fourth embodiment, and acquires the polarization information inS1403 similar to S1302 of the fourth embodiment. In S1404, thedifference in the phase component θ is determined in the same manner asS1303 of the fourth embodiment, the final representative wavelength isdetermined, and the combined image is created in S1405. Therepresentative wavelength for each color channel may be the same for allthe divided areas.

Sixth Embodiment

A flowchart in FIG. 15 shows image processing executed by an imageprocessing apparatus 1200 according to a sixth embodiment of thedisclosure. In FIG. 15 , S1500 to S1503 and S1505 are the same as S1300to S1304 of the fourth embodiment (FIG. 13 ).

In this embodiment, in S1504 before the combined image is generated inS1505, the image processing apparatus 1200 detects an area estimated tobe white using the finally determined representative wavelength, andcalculates a white balance correction coefficient. In S1505, the imageprocessing apparatus 1200 creates a combined image and performs thewhite balance correction for the combined image using the white balancecorrection coefficient.

The image may be divided into areas (blocks) between S1500 and S1501similar to S1401 of the fifth embodiment. At this time, when thecaptured image is divided into a plurality of blocks in calculating thewhite balance correction coefficient, the area division of the imageperformed in advance may be shared.

Seventh Embodiment

FIG. 17 shows a configuration of an image pickup apparatus 1700 thatincludes the image processing apparatus 1200 according to one of thefirst to third embodiments. The image pickup apparatus 1700 includes animaging lens 1701 that images light from an object, and an image sensor1702 such as a CCD sensor or a CMOS sensor that captures(photoelectrically converts) an object image formed by the imaging lens1701. A polarization acquirer 1710 is disposed on the object side of theimaging lens 1701. The polarization acquirer 1710 may be disposedbetween the imaging lens 1701 and the image sensor 1702.

The polarization acquirer 1710 includes a waveplate (quarter waveplate)1703, a variable retardation plate 1704, and a polarizing plate 1705.The quarter waveplate 1703 provides a relative phase difference(retardation) of π/2 between the orthogonal polarization components ofthe incident light. This embodiment uses the quarter waveplate for theretardation plate but may use three-quarter waveplate or the like aslong as the relative phase difference of π/2 can be imparted.

Similar to the quarter waveplate 1703, the variable retardation plate1704 can give a relative phase difference between the orthogonalpolarization components of the incident light, and can further changethe given relative phase difference. The variable retardation plate 1704can use one that can change the phase difference using a liquid crystal.The phase difference of the variable retardation plate 1704 is variablyset according to the voltage applied from a phase difference setter1706. The polarizing plate 1705 transmits a component in thetransmission axis direction (transmission polarization direction) amongthe polarization components of the incident light.

The polarization acquirer 1710 can change the polarization angle of thelight that has transmitted through the polarization acquirer 1710, bychanging the phase difference given by the variable retardation plate1704. Thus, by imaging a plurality of times by changing the phasedifference given by the variable retardation plate 1704 in capturing theimage, the image pickup apparatus 1700 can generate a plurality ofimages by capturing a plurality of polarized light beams havingdifferent polarization states.

The image pickup apparatus 1700 has the image processing apparatus 1200shown in FIG. 1 that acquires a plurality of captured images generatedwith the imaging signals from the image sensor 1702. As described in thefourth to sixth embodiments, the image processing apparatus 1200acquires the polarization information from the acquired plurality ofcaptured images, and uses them to generate and output (record ordisplay) a combined image.

Other Embodiments

Embodiment(s) of the disclosure can also be realized by a computer of asystem or apparatus that reads out and executes computer executableinstructions (e.g., one or more programs) recorded on a storage medium(which may also be referred to more fully as a ‘non-transitorycomputer-readable storage medium’) to perform the functions of one ormore of the above-described embodiment(s) and/or that includes one ormore circuits (e.g., application specific integrated circuit (ASIC)) forperforming the functions of one or more of the above-describedembodiment(s), and by a method performed by the computer of the systemor apparatus by, for example, reading out and executing the computerexecutable instructions from the storage medium to perform the functionsof one or more of the above-described embodiment(s) and/or controllingthe one or more circuits to perform the functions of one or more of theabove-described embodiment(s). The computer may comprise one or moreprocessors (e.g., central processing unit (CPU), micro processing unit(MPU)) and may include a network of separate computers or separateprocessors to read out and execute the computer executable instructions.The computer executable instructions may be provided to the computer,for example, from a network or the storage medium. The storage mediummay include, for example, one or more of a hard disk, a random-accessmemory (RAM), a read only memory (ROM), a storage of distributedcomputing systems, an optical disk (such as a compact disc (CD), digitalversatile disc (DVD), or Blu-ray Disc (BD)™), a flash memory device, amemory card, and the like.

When an output image different from the input image is generated, theabove embodiments can reduce coloring of the output image.

While the disclosure has been described with reference to exemplaryembodiments, it is to be understood that the disclosure is not limitedto the disclosed exemplary embodiments. The scope of the followingclaims is to be accorded the broadest interpretation so as to encompassall such modifications and equivalent structures and functions.

This application claims the benefit of Japanese Patent Application Nos.2020-116306, filed on Jul. 6, 2020 and 2020-160360, filed on Sep. 25,2020 which are hereby incorporated by reference herein in theirentirety.

What is claimed is:
 1. An apparatus comprising a memory storinginstructions, and at least one processor that, executes the instructionsto: acquire a plurality of input images obtained via image capturingusing light with a plurality of different polarization states; angleinformation on each of first pixels corresponding to a first wavelengthin each of the input images based on a first luminance value of each ofthe first pixels; acquire second angle information on each of secondpixels corresponding to a second wavelength in a specific area in eachof the input images based on a second luminance value of each of thesecond pixels; acquire corrected polarization information obtained byreplacing the first angle information on at least one pixel in thespecific area among the first pixels with the second angle information;and generate an output image using the corrected polarizationinformation, wherein the first angle information represents a firstpolarization angle that maximizes the first luminance value, and whereinthe second angle information represents a second polarization angle thatmaximizes the second luminance value.
 2. The apparatus according toclaim 1, wherein the at least one processor executes the instructions toacquire third angle information on each of third pixels corresponding toa third wavelength in each of the input images based on a luminancevalue of each of the third pixels, and wherein the correctedpolarization information obtained by replacing the third angleinformation on at least one pixel in the specific area among the thirdpixels with the second angle information.
 3. The apparatus according toclaim 1, wherein the first wavelength and the second wavelength arewavelengths corresponding to any two of R, G and B.
 4. The apparatusaccording to claim 2, wherein the first wavelength, the secondwavelength and the third wavelength are wavelengths respectivelycorresponding to any of R, G and B.
 5. The apparatus according to claim1, wherein the at least one processor executes the instructions toselects, as the specific area, an area in each of the input images inwhich a degree of polarization is larger than a predetermined value. 6.The apparatus according to claim 1, wherein the at least one processorexecutes the instructions to selects, as the specific area, an area inwhich a difference in the first polarization angle and secondpolarization angle is larger than a predetermined value.
 7. Theapparatus according to claim 1, wherein the at least one processorexecutes the instructions to divides each of the input images into aplurality of areas based on wavelength information, and to selects thespecific area from the plurality of areas.
 8. The apparatus according toclaim 1, wherein the at least one processor executes the instructions toselects the second wavelength according to polarization information orwavelength information in the specific area.
 9. An image pickupapparatus comprising: the apparatus according to claim 1; and a sensorconfigured to acquire input images.
 10. The image pickup apparatusaccording to claim 9, further comprising a polarizing element configuredto control a polarization state of transmitting light.
 11. The imagepickup apparatus according to claim 10, wherein the polarizing elementincludes a variable retardation plate made of liquid crystal, and apolarizing plate.